{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification (2) – implementation and application of Nearest Neighbour classification, and Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we continue on with some of methods of \n",
    "classification, starting with an implementation of Naive Bayes, then an application of Naive Bayes on a benchmark dataset. The notebook also looks into the related method of Logistic Regression for comparison."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "Classification models learn a mapping $h(\\boldsymbol{X})$ from a feature space $\\boldsymbol{X}$ to a finite set of labels $\\boldsymbol{Y}$\n",
    "\n",
    "\n",
    "In this lab we will focus for simplicity on binary classification, where the labels are assumed to be in $\\{-1,1\\}$ or alternatively $\\{0,1\\}$. \n",
    "\n",
    "\n",
    "We will use simple generated datasets and a real data set on the sinking of the Titanic to explore some different classification algorithms. For a description of the variables and more information on the data see: https://www.kaggle.com/c/titanic-gettingStarted/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n",
      "SKLEARN 0.20.1\n",
      "SCIPY 1.1.0\n",
      "NUMPY 1.15.4\n",
      "MATPLOTLIB 3.0.2\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "%load_ext autoreload\n",
    "%autoreload\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as pl\n",
    "import util\n",
    "\n",
    "from scipy.stats import itemfreq\n",
    "from scipy.stats import bernoulli\n",
    "from scipy.stats import multivariate_normal as mvnorm\n",
    "\n",
    "import sklearn\n",
    "import scipy\n",
    "import matplotlib\n",
    "print(\"SKLEARN\",sklearn.__version__)\n",
    "print (\"SCIPY\",scipy.version.full_version)\n",
    "print(\"NUMPY\",np.__version__)\n",
    "print(\"MATPLOTLIB\",matplotlib.__version__)\n",
    "\n",
    "X, Y = util.load_data() # passenger_class, is_female, sibsp, parch, fare, embarked (categorical 0-3)\n",
    "X_demean = X - np.mean(X, axis=0)\n",
    "X_unitsd = X_demean/(np.std(X_demean,axis=0))\n",
    "X_whiten = np.dot(X_demean, util.whitening_matrix(X_demean))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One approach to learning a classification function $h(\\boldsymbol{X})$ is to model $P(y|\\boldsymbol{x})$ and convert that to a classification by setting:\n",
    "\\begin{equation}h(\\boldsymbol{X}) = \\begin{cases} 1 & \\text{if }P(y|\\boldsymbol{x}) > \\frac{1}{2}\\\\ 0 & \\text{otherwise}\\end{cases}\n",
    "\\end{equation}\n",
    "\n",
    "\n",
    "Example: Suppose we want to build a model to predict the probability of survival on the Titanic based on just two categorical features, a persons class (1,2 or 3) and their sex (1=female,0=male). An obvious approach would be to create a category for each combination of our features (female 1st, female 2nd ... male 3rd) and calculate the proportion who survived in each as an estimate for the survival probability $P(y|\\boldsymbol{x})$. For each observation in our test data - we simply look up the survival rate in the corresponding category.\n",
    "\n",
    "This corresponds to maximum likelihood estimation: $\\hat{\\theta} = argmax_{\\theta'}P(data|\\theta')$, where the parameters, $\\theta$, we want to estimate are the true probabilities $P(y|\\boldsymbol{x})$ for each combination of $\\boldsymbol{x}$ and $data$ is the set features and labels we have observed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 0) 0.38961038961038963\n",
      "(1, 1) 0.9846153846153847\n",
      "(2, 0) 0.1780821917808219\n",
      "(2, 1) 0.9122807017543859\n",
      "(3, 0) 0.14218009478672985\n",
      "(3, 1) 0.4583333333333333\n"
     ]
    }
   ],
   "source": [
    "combinations = [(i,j) for i in [1,2,3] for j in [0,1]]\n",
    "for c in combinations:\n",
    "    match = np.where((X[:,0] == c[0]) * (X[:,1] == c[1]))[0]\n",
    "    print(c,sum(Y[match])/float(len(match)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** *Why will this approach not work in general? What happens as we increase the number of features or the number of values each feature can take? What about if features are continuous?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "\n",
    "\n",
    "Following Bayes Rule, we can write:\n",
    "\n",
    "$P(y|\\boldsymbol{x}) \\propto P(y)P(\\boldsymbol{x}|y) = P(y)P(x_1,x_2...x_D|y) $\n",
    "\n",
    "It easy to estimate $P(y)$ as just the proportions of each class in the training data. We could also directly estimate $P(x_1,x_2...x_D|y)$ for each $y$ (for example with kernel density estimation) but as the number of features $D$ gets large this estimation suffers the curse of dimensionality.\n",
    "\n",
    "Naive Bayes assumes that the data was generated by a model where all the features are independent of one-another given the class label so that we can estimate $P(x_j|y)$ separately for each feature.\n",
    "\n",
    "\\begin{equation}\n",
    "P(y|\\boldsymbol{x}) \\propto P(y)\\prod_{j=1}^D P(x_j|y)\n",
    "\\end{equation}\n",
    "\n",
    "The normalisation constant can be obtained;\n",
    "\n",
    "\\begin{equation}\n",
    "P(y|\\boldsymbol{x}) = \\frac{P(y)\\prod_{j=1}^D P(x_j|y)}{P(\\boldsymbol{x})},\n",
    "\\end{equation}\n",
    "where,\n",
    "\\begin{equation}\n",
    "P(\\boldsymbol{x}) = P(y=0)\\prod_{j=1}^D P(x_j|y=0) + P(y=1)\\prod_{j=1}^D P(x_j|y=1),\n",
    "\\end{equation}\n",
    "this operation is called [marginalisation](http://en.wikipedia.org/wiki/Marginal_distribution), since we marginalise (or sum/integrate out) $y$ from the joint distribution (top line) $P(y, \\mathbf{x}) = P(y)P(\\mathbf{x}|y)$ to obtain a distribution over $\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** *Implement a Naive Bayes model for the Titanic data set using passenger_class and is_female as features*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-5901c36ade95>, line 19)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-5901c36ade95>\"\u001b[0;36m, line \u001b[0;32m19\u001b[0m\n\u001b[0;31m    self.gs = YOUR CODE HERE # return a map from gender value to probability: Hint the proportions function above may help\u001b[0m\n\u001b[0m                      ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# a function that may be useful\n",
    "def proportions(array):\n",
    "    \"\"\" returns a map from each unique value in the input array to the proportion of times that value occures \"\"\"\n",
    "    prop = itemfreq(array)\n",
    "    prop[:,1] = prop[:,1]/sum(prop,axis=0)[1]\n",
    "    return dict(prop)\n",
    "\n",
    "class Naive_Bayes:\n",
    "    def train(self,X,Y):\n",
    "        \"\"\" trains the model with features X and labels Y \"\"\"\n",
    "        # 1) Estimate P(Y=1)\n",
    "        self.py = sum(Y)/float(len(Y))\n",
    "\n",
    "        # 2) For each feature, x, estimate P(x|y=1) and P(x|y=0)\n",
    "        survived = X[np.where(Y==1)[0],:] # the features of those who survived\n",
    "        died  = X[np.where(Y==0)[0],:] # the features for those who died\n",
    "\n",
    "        # estimate P(gender|survived)\n",
    "        self.gs = YOUR CODE HERE # return a map from gender value to probability: Hint the proportions function above may help\n",
    "\n",
    "        # estimate P(class|survived)\n",
    "        self.cs = YOUR CODE HERE # return a map from class to probability for those who survived\n",
    "\n",
    "        # estimate P(gender|died)\n",
    "        self.gd = YOUR CODE HERE # return a map from gender value to probability for those who died\n",
    "\n",
    "        # estimate P(class|died)  \n",
    "        self.cd = YOUR CODE HERE # return a map from class to probability for those who died\n",
    "    \n",
    "    def predict(self,sex,p_class):\n",
    "        \"\"\" outputs the probability of survival for a given class and gender \"\"\"\n",
    "        # caclulate unormalized P(y = 1|sex,p_class) as P(y=1)P(sex|y=1)P(p_class|y=1) \n",
    "        ps = YOUR CODE HERE\n",
    "\n",
    "        # calculate unormalized P(y = 0|sex,p_class) as P(y=0)P(sex|y=0)P(p_class|y=0)\n",
    "        pd = YOUR CODE HERE\n",
    "\n",
    "        # calculates the survival ratio as ps/pd and the normalized probability  from the ratio\n",
    "        r = ps/pd\n",
    "        psn = r/(1+r)\n",
    "        return psn\n",
    "\n",
    "# run the model\n",
    "model = Naive_Bayes()\n",
    "model.train(X,Y)\n",
    "for p_class,sex in combinations:\n",
    "    print((p_class,sex),model.predict(sex,p_class))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** *Compare these predictions with those just based on the proportion of survivals. How true is the Naive Bayes assumption for this case?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** *How does the number of parameters to be learnt scale with the number of features for Naive Bayes?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** *Run Naive Bayes from Sci-Kit Learn using the same features.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "# Sklearn doesn't have a model that expects categorical data. \n",
    "# We need to first encode our (p_class, sex) to (is_first,is_second,is_third,is_female,is_male)\n",
    "\n",
    "# use preprocessing.OneHotEncoder to create a new dataset X2 that is the transformation of the first 2 columns of X\n",
    "\n",
    "nb_enc = OneHotEncoder()\n",
    "X2 = # use the encoder to transform the first two columns of X \n",
    "\n",
    "# fit a Multinommial Naive Bayes Model\n",
    " \n",
    "\n",
    "\n",
    "# transforms our combinations to the one-hot encoding\n",
    "c = nb_enc.transform(np.asarray(combinations)).toarray()\n",
    "\n",
    "# gets predictions for each combination\n",
    "predictions = nb.predict_proba(c)\n",
    "\n",
    "# prints your predictions in the same format as previous models\n",
    "for i in range(len(c)):\n",
    "    print(combinations[i],predictions[i][1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naive Bayes can also handle continuous features. The data below is generated by a ```Gaussian mixture model```. For each class there is a separate 2-dimensional Gaussian distribution over the features x1, x2. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generates some data from a Gaussian Mixture Model. \n",
    "mean0 = [-1,-1]  # the mean of the gaussian for class 0      \n",
    "mean1 = [1,1] # the mean of the gaussian for class 1\n",
    "cov0 = [[.5, .28], [.28, .5]] # the covariance matrix for class 0\n",
    "cov1 = [[1, -.8], [-.8, 1]] # the covariance matrix for class 1\n",
    "mixture = util.GaussianMixture(mean0,cov0,mean1,cov1)\n",
    "mX,mY = mixture.sample(500,0.5,plot=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** *Fit a Gaussian Naive Bayes model using Sklearn*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "# fit a GaussianNB model\n",
    "\n",
    "gnb = YOUR CODE HERE # create and fit a Gaussian Naive Bayes model to the Gaussian mixture data mX,mY\n",
    "\n",
    "\n",
    "# plots the probability that a point in x,y belogs to the class Y=1 according to your model and the decision boundary p=.5\n",
    "x = np.linspace(-4,4,100)\n",
    "y = np.linspace(-4,4,100)\n",
    "test_points = np.dstack(np.meshgrid(x, y)).reshape(-1,2)\n",
    "z = gnb.predict_proba(test_points)[:,1].reshape(len(x),len(y)) # probability Y = 1\n",
    "f,ax = subplots(1,1,figsize=(5,5))\n",
    "cn = ax.contourf(x,y,z)\n",
    "ct = ax.contour(cn,levels=[0.5])\n",
    "ax.scatter(mX[:,0],mX[:,1],s=5, c = [\"black\" if t < 1 else \"white\" for t in mY],alpha=1)\n",
    "ax.clabel(ct)\n",
    "show()\n",
    "\n",
    "# Try changing the covariance matrices and refitting your model. \n",
    "# When does the probability distribution returned by Naive Bayes resemble the true one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "Logistic regression models $P(y|\\boldsymbol{x})$ directly by assuming it is a (logistic) function of a linear combination of the features. The logistic function $\\theta(s) = \\frac{e^s}{e^s+1}$ maps the weighted features to $[0,1]$ to allow it to model a probability. Training logistic regression corresponds to learning the weights $\\boldsymbol{w}$ to maximise the likelihood function:\n",
    "\n",
    "\\begin{equation}\n",
    "P(y_1...y_n|\\boldsymbol{x}_1...\\boldsymbol{x}_n,\\boldsymbol{w}) = \\prod_{i=1}^n \\theta(y_i\\boldsymbol{w}^T\\boldsymbol{x}_i)\n",
    "\\end{equation}\n",
    "\n",
    "Maximising the likelihood $P(y_1...y_n|\\boldsymbol{x}_1...\\boldsymbol{x}_n,\\boldsymbol{w})$ is equivalent to minimising the negative log-likelihood: \n",
    "\\begin{equation}\n",
    "\\boldsymbol{w}^* = argmin_{\\boldsymbol{w}}\\left( -\\log\\left(\\prod_{i=1}^n \\theta(y_i\\boldsymbol{w}^T\\boldsymbol{x}_i)\\right)\\right)\n",
    "= argmin_{\\boldsymbol{w}}\\left( \\sum_{i=1}^n \\ln(1+e^{-y_i\\boldsymbol{w}^T\\boldsymbol{x}_i})\\right)\n",
    "\\end{equation}\n",
    "\n",
    "Once we have the weights $\\boldsymbol{w}^*$, we can predict the probability that a new observation belongs to each class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** *Suppose that we have a data set that is linearly separable. What happens to the weights $w$ when we run linear regression?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** *Use Sklearn to fit a logistic regression model on the Gaussian mixture data.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run Logistic regression on the gaussian mixture data\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logistic = YOUR CODE HERE # create and fit a logistic regression model on mX,mY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the probability y = 1 as over the feature space as for Naive Bayes\n",
    "logistz = logistic.predict_proba(test_points)[:,1].reshape(len(x),len(y)) # probability Y = 1\n",
    "f,ax = subplots(1,1,figsize=(5,5))\n",
    "cn = ax.contourf(x,y,logistz)\n",
    "ct = ax.contour(cn,levels=[0.5])\n",
    "ax.scatter(mX[:,0],mX[:,1],s=5, c = [\"black\" if t < 1 else \"white\" for t in mY],alpha=1)\n",
    "ax.clabel(ct)\n",
    "show()# implement the jacobian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can model more complex decision boundaries by expanding the feature space to include combinations of features\n",
    "\n",
    "# re-fit logistic regression adding in all quadratic combinations of features ie x1,x2,x1x2,x1^2,x2^2\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "poly_expand = YOUR CODE HERE # create a polynomial feature transformer that produces quadratic combinations\n",
    "m2X = YOUR CODE HERE # use poly_expand to transform the original features (mX)\n",
    "logistic.YOUR CODE HERE # fit the logistic model with the new features\n",
    "\n",
    "# transform the test plots and predict and plot\n",
    "testpoints2 = poly_expand.transform(test_points)\n",
    "logistic2z = logistic.predict_proba(testpoints2)[:,1].reshape(len(x),len(y)) # probability Y = 1\n",
    "f,ax = subplots(1,1,figsize=(5,5))\n",
    "cn = ax.contourf(x,y,logistic2z)\n",
    "ct = ax.contour(cn,levels=[0.5])\n",
    "ax.scatter(mX[:,0],mX[:,1],s=5, c = [\"black\" if t < 1 else \"white\" for t in mY],alpha=1)\n",
    "ax.clabel(ct)\n",
    "show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With large numbers of features there is a risk of overfitting to the training data. We can tune a logistic regression model to reduce the risk of overfitting by penalising large weights, $\\boldsymbol{w}$ \n",
    "\n",
    "**Exercise:** *Experiment with the regularisation parameters sklearn provides: \n",
    "penalty = \"l1\" or \"l2\" and C = inverse of weight of regularisation term.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lreg = YOUR CODE HERE # create and fit a regularized logistic regression model to the quadraticly expanded features\n",
    "\n",
    "\n",
    "# plots the probability as before\n",
    "logistic2z_reg = lreg.predict_proba(testpoints2)[:,1].reshape(len(x),len(y)) # probability Y = 1\n",
    "f,ax = subplots(1,1,figsize=(5,5))\n",
    "cn = ax.contourf(x,y,logistic2z_reg)\n",
    "ct = ax.contour(cn,levels=[0.5])\n",
    "ax.scatter(mX[:,0],mX[:,1],s=5, c = [\"black\" if t < 1 else \"white\" for t in mY],alpha=1)\n",
    "ax.clabel(ct)\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run logistic regression on the titanic data\n",
    "\n",
    "titanic_logist = YOUR CODE HERE # create and fit a logistic regression model on the titanic data\n",
    "\n",
    "\n",
    "# Look at the coefficients (weights) in the model. Are they meaningfull? \n",
    "# Do you need to change the way any of the features were encoded?\n",
    "print(titanic_logist.coef_)\n",
    "print(titanic_logist.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement Logistic Regression (Optional)\n",
    "Recall for logistic regression we are trying to find (assuming we have encoded $Y$ as $\\{-1,1\\}$)  \n",
    "\\begin{equation}\n",
    "\\boldsymbol{w}^* = argmin_{\\boldsymbol{w}}\\left( \\sum_{i=1}^n \\ln(1+e^{-y_i\\boldsymbol{w}^T\\boldsymbol{x}_i})\\right)\n",
    "\\end{equation}\n",
    "\n",
    "This is a convex optimisation problem in $\\boldsymbol{w}$.\n",
    "\n",
    "We can solve it using gradient decent (or an optimisation library)\n",
    "\n",
    "**Exercise:** *Implement logistic regression using scipy's optimisation library and run it on the Gaussian mixture data mX,mY.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implement logistic regression using the scipy's optimization library and run it on the gaussian model data\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "# copy input so our modifications don't effect original data\n",
    "dataX = mX\n",
    "dataY = mY\n",
    "\n",
    "# encode mY as -1,1\n",
    "dataY[mY==0] = -1\n",
    "\n",
    "# add a column of all ones to mX to allow us to fit an intercept\n",
    "dataX = np.hstack((np.ones((mX.shape[0],1)),mX))\n",
    "\n",
    "\n",
    "# implement the loss function\n",
    "def loss(w,X,Y):\n",
    "    YOUR CODE HERE \n",
    "    \n",
    "\n",
    "# start the optimization with randomly guessed weights    \n",
    "w0 = np.random.random((dataX.shape[1],1)) \n",
    "\n",
    "# runs the optimisation\n",
    "optimal = minimize(loss,w0,args=(dataX,dataY),method=\"BFGS\")    \n",
    "w = optimal.x\n",
    "print(w)\n",
    "\n",
    "# how does this compare with the coefficients you saw using Sklearn? \n",
    "# try refitting the sklearn logistic model with a very high value for C (like 10000)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimisation method we are using (BFGS) needs to know the jacobian (gradient of the loss function with respect to w). Since we didn't supply it, python is approximating it numerically. We can speed things up by supplying it.\n",
    "\n",
    "\\begin{equation}\n",
    "L(\\boldsymbol{w}) = \\sum_{i=1}^n \\ln(1+e^{-y_i\\boldsymbol{w}^T\\boldsymbol{x}_i})  \\longleftarrow \\text{ loss function}\\\\\n",
    "\\nabla{L} = [\\frac{\\partial L}{\\partial w_1},\\frac{\\partial L}{\\partial w_2}, ..., \\frac{\\partial L}{\\partial w_D}] \\longleftarrow \\text{ definition of gradient}\\\\\n",
    "\\frac{\\partial L}{\\partial w_j} = -\\sum_{i=1}^n x_{ij} \\frac{ y_i e^{-y_i\\boldsymbol{w}^T\\boldsymbol{x}_i}}{1+e^{-y_i\\boldsymbol{w}^T\\boldsymbol{x}_i}} \\longleftarrow \\text{ result of taking partial derivative of loss function with respect to weight $j$}\\\\\n",
    "\\end{equation}\n",
    "\n",
    "**Exercise:** *repeat the previous exercise but supply the Jacobian to the minimizer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implement jacobian\n",
    "\n",
    "def grad_loss(w,X,Y):\n",
    "    YOUR CODE HERE\n",
    "    \n",
    "# start the optimization with randomly guessed weights    \n",
    "w0 = np.random.random((dataX.shape[1],1)) \n",
    "\n",
    "optimal = minimize(loss,w0,args=(dataX,dataY),jac = grad_loss, method=\"BFGS\")    \n",
    "print(optimal)\n",
    "w = optimal.x\n",
    "print(w) \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
